Skip to main content

Reply to this post | Go Back
View Post [edit]

Poster: SethPen Date: Oct 1, 2013 9:19am
Forum: web Subject: "retroactive and permanent"


I have a site that I like to view from time to time and I am concerned, with the current rules, it may not be available for future individuals. (Page cannot be crawled or displayed due to robots.txt.)

From your Frequently Asked Questions:

" Currently there is no way to exclude only a portion of a site, or to exclude archiving a site for a particular time period only. When a URL has been excluded at direct owner request from being archived, that exclusion is retroactive and permanent. "

Previously, the owner wanted this to be Archived:

(This is an online version of a book called "Portal: A Dataspace Retrieval")

"The pages are designed to be search engine friendly – should a crawler bump into one of the pages the entire site should be absorbed in no time at all. I hope people will soon start bumping into theses pages at random, and then perhaps start reading this odd story. "

Am I to understand correctly from your Frequently Asked Questions, that if a site posts a new robot.txt with the correct parameters to not be index, it will be retroactive? Including previous years?

Will this ever change?

Additionally I want to thank you for a great service and I have been amazed with what you have been able to archive.